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PREFACE 


The  research  described  in  this  report  was  conducted  at  the  Armstrong 
Laboratory,  Human  Resources  Directorate,  Aircrew  Training  Research  Division 
(AL/HRA),  in  Mesa,  Arizona,  by  the  University  of  Dayton  Research  Institute  (UDRI). 

AL/HRA  conducts  visual  training  effectiveness  research  in  support  of  aircrew 
training  technology.  This  research,  performed  by  UDRI,  describes  a  US  Air  Force 
training  program  for  visualizing  large-scale  dynamic  spatial  displays.  The  Virtual 
Environment  Debrief  Interface  (VirDI)  shows  how  virtual  environment  technologies 
can  be  applied  to  an  aircrew  training  problem.  * 

This  effort  was  conducted  under  Work  Unit  1123-03-85,  Flying  Training 
Research  Support,  Contract  No.  F33615-90-C-0005.  The  Laboratory  Contract 
Monitor  was  Ms  Patricia  A.  Spears;  the  Laboratory  Task  Order  Monitor  was  Dr 
Richard  A.  Thurman. 
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VISUALIZING  SPATIAL  RELATIONSHIPS: 

TRAINING  FIGHTER  PILOTS  IN  A  VIRTUAL  ENVIRONMENT  DEBRIEF  INTERFACE 


INTRODUCTION 

Military  and  nonmilitary  tasks  alike  often  require  the  individual  to  acquire  and  maintain 
dynamic  spatial  awareness— that  component  of  situational  awareness  which  involves  the 
ability  to  conceptualize  the  dynamic  location  of  multiple  objects  in  three-dimensional  space. 
Under  normal  flying  conditions,  the  out-the-window  view  usually  provides  sufficient  visual 
information  to  maintain  spatial  awareness.  However,  there  are  many  tasks  which  require 
the  operator  to  attribute  a  logical  set  of  spatial  properties  to  unseen  objects.  For  example, 
during  an  air  intercept  a  fighter  pilot  plans  most  tactical  maneuvers  well  before  acquiring 
visual  contact  with  the  target.  Success  depends  critically  upon  the  ability  to  construct  an 
internal  representation,  or  mental  model,  of  the  problem  space  from  which  one  can  define 
or  constrain  the  situation,  and  plan,  test,  and  evaluate  potential  solutions.  This  would 
suggest  that  systems  designed  to  train  personnel  for  spatial  awareness  should  address  the 
perceptual  and  cognitive  processes  involved  In  the  creation  and  use  of  mental  models. 

One  of  the  greatest  challenges  for  developers  of  aircrew  training  systems,  however,  is  to 
present  information  to  the  trainee  in  a  manner  that  will  engage  these  innate  human 
capacities. 

This  report  describes  the  goals,  capabilities,  and  constraints  of  a  virtual  world 
visualization  tool  developed  for  the  Air  Force  at  the  Armstrong  Laboratory,  Aircrew  Training 
Research  Division  (AL/HRA),  in  Mesa,  AZ.  The  Virtual  Environment  Debrief  Interface 
(VirDI)  is  a  portable  post-mission  feedback  tool  designed  to  allow  pilots  to  study  beyond- 
visual-range  air  intercept  performance  in  a  three-dimensional  synthetic  environment.  There 
are  alternative  post-mission  feedback  systems  currently  available,  but  the  graphics  typically 
consist  of  two-dimensional  representations  of  the  airspace  (e.g.,  the  “god’s-eye  view”). 
Although  two-dimensional  images  can  enhance  the  training  environment,  subject  matter 
experts  continue  to  express  the  need  for  a  three-dimensional  visualization  tool  that  would 
represent  directly  the  intercept  space.  Recent  developments  in  computer  simulation  and 
human/computer  interface  methods  and  technologies  have  provided  the  vehicle  for  the 
development  of  VirDI — a  feedback  system  in  which  the  trainee  is  immersed  in  an 
interactive  three-dimensional  virtual  world  of  visual  imagery. 

Spatial  Awareness  Training  in  a  Virtual  World 

Immersive  virtual  environments  are  defined  as  a  coordinated  set  of  computer,  graphic, 
and  synthetic  sensory  input/output  technologies  and  techniques  to  create  in  the  user  an 
illusion  of  Immersion  In  a  computer-generated  reality.  ‘The  user  interacts  with  a  world 
containing  seemingly  real,  three-dimensional  objects  in  three-dimensional  space  that 
respond  interactively  to  each  other  and  the  user"  (Bryson,  1992,  p.  1.1).  While  computer¬ 
generated  “realities”  are  synthetic,  the  psychological  impact  of  the  immersive  experience 
may  be  very  powerful.  The  user  no  longer  feels  that  (s)he  is  interacting  with  a  picture  of  a 
thing,  as  on  a  video  monitor;  rather,  the  experience  is  direct  and  intuitive — as  /fit  were 
unfolding  in  the  same  world  that  envelops  the  user. 
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By  providing  an  interactive  synthetic  world,  virtual  environments  have  considerable 
potential  as  training  media  for  aiding  fighter  pilots  in  visualizing  beyond-visual-range  air 
intercepts  (cf.  Ambum  &  Stytz,  1992;  Ellis,  1991a,  b;  Zeltzer  &  Drucker,  1992).  The 
training  medium  provides  a  perceptual  experience  of  data  that  otherwise  could  not  be 
achieved  in  the  real  world.  Moreover,  the  user  can  interact  with  the  world  in  a  natural  and 
intuitive  fashion  as  if  interacting  with  objects  in  the  real  world.  In  a  synthetic  world,  normal 
human  capacities  also  can  be  augmented  to  endow  the  user  with  super-human  control. 
Sensory-motor  barriers  can  be  breached,  so  the  user  also  can  walk  through  large  expanses 
of  airspace  in  a  few  steps,  viewing  the  mission  from  multiple  vantage  points.  It  is 
anticipated  that  the  virtual  world  experience  also  may  serve  to  provoke  or  augment  mental 
modeling  in  the  student  pilot.  It  is  possible  that  such  a  system  will  facilitate  the  transition 
from  the  student’s  meager  mental  model  of  the  air  intercept  problem  to  the  well-articulated, 
efficient  mental  model  characteristic  of  the  expert  tighter  pilot. 

THE  VIRTUAL  ENVIRONMENT  DEBRIEF  INTERFACE 

The  dual  functions  of  the  Virtual  Environment  Debriet  Interface  are:  (a)  to  stimulate  the 
senses  with  perceptual  re-presentation  of  a  data  set  containing  the  spatio-temporal 
parameters  of  an  air  intercept  and ,  (b)  to  provide  a  template  tor  the  student  to  construct  a 
mental  image  of  air  intercept  space.  In  addition  to  these  two  functions,  the  requirements  of 
this  project  focused  on  developing  a  visualization  tool  that,  it  successful,  could  be  deployed 
with  various  training  and  operational  flight  squadrons.  Thus,  the  VirDI  system  has  been 
designed  to  be  user-friendly,  low  cost  and  portable,  with  minimal  installation  or 
maintenance  costs.  It  consists  of  an  image  generating  system  controlled  by  a  386  personal 
computer  and  commercially  available  helmet-mounted  display  and  head  tracking  devices. 
The  remaining  sections  describe  the  system  in  greater  detail  and  provide  an  account  of  its 
training  potential  when  deployed  at  the  squadron  level. 

System  Design 

The  VirDI  system  consists  of  a  coordinated  set  of  input/output  devices  controlled  by  an 
XTAR  FALCON-PC™  25  MHz  386  personal  computer.  The  XTAR  system  contains: 

1)  2  XTAR  AP-2020  Array  Processor^^  boards,  which  perform  the  database  transfor¬ 
mations  and  deliver  45,000  50  x  50-plxel  polygons  per  cnannel  per  second; 

2)  2  XTAR  PG-2000™  graphics  boards,  which  project  the  images  at  a  pixel  fill  rate  of 
1 60  million  pixels  per  channel  per  second; 

3)  2  XTAR  GL-2000™  GenLOCK  boards  which  generate  the  NTSC  graphics  signal 
to  the  helmet-mounted  display. 

A  set  of  boards  (1  AP-2020  board,  1  PG-2000  board  and  1  GL-2000  board)  is  needed 
for  each  of  two  channels  (lett/right  eye)  to  achieve  a  stereoscopic  display.  The  pilot 
observes  the  intercept  airspace  using  a  Flight  Helmet™ ,  a  stereoscopic  helmet-mounted 
display.  1  his  display  is  composed  of  a  pair  of  color  LCD  displays  (360  x  240  resolution) 
viewed  through  LEEP™  wide-angle  optics.  The  combined  images  seen  through  these 
optics  produce  a  90°  (h)  x  30°  (v)  (approx.)  field  of  view.  The  pilot  has  unrestricted 
movement  through  the  virtual  environment  using  a  head-tracking  device  provided  by  the 
extended  range  Flock  of  Birds™  (sensitive  over  an  fHoot  radius).  Alternatively,  a  seated 
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user  may  maneuver  through  the  virtual  world  using  a  6<iegree-of-freedom  Spacebali™ 

(Fig.  1).  A  Cricket  3D™  input  device  allows  the  user  to  touch,  select,  and  move  objects 
about  in  the  virtual  world  with  tactile  feedback.  Finally,  the  helmet  also  is  equipped  with  a 
stereophonic  headset  for  inputting  sound  while  the  user  experiences  the  virtual 
environment.  At  present,  this  capability  is  exploited  with  a  simple  audio  cassette  to  provide 
prerecorded  auditory  feedback.  Any  user  familiar  with  computers,  simulators,  or  video 
arcade  games  can  readily  adapt  to  using  the  virtual  environment  interface. 


Figure  1 

Overview  of  the  VirDI  Post-Mission  Feedback  System. 

Stereoscopic  images  are  projected  on  a  pair  of  helmet-mounted  LCDs.  The  imagery  is  slaved 
tothe  pilofs  head  movements  based  on  input  from  a  heacKracking  system.  As  shown  here,  the 
pilot  also  may  nrrave  through  the  world  using  a  6-degree-of-freedom  input  device. 
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The  VirDI  system  is  designed  to  interface  with  the  existing  F-16  Air  Intercept  Trainer 
(AIT)’  at  Williams  AFB.  The  AIT  is  a  part-task  training  system  designed  to  help  fighter 
pilots  define  and  implement  air  intercept  strategies  (Fig.  2).  It  offers  concentrated  practice 
on  the  symbology  and  switchology  of  the  fighter  aircraft  under  simulated  air  combat 
conditions.  At  present,  88  training  scenarios  are  available,  incorporating  single  and 
multiple-ship  situations  as  well  as  maneuvering  or  nonmaneuvering  targets.  Throughout 
the  simulated  sortie,  the  AIT  provides  real-time  simulation  of  the  mission-critical 
displays  found  in  the  F-16  cockpit,  including  the  radar  electro-optical  (REO)  display, 
multi-function  display  (MFD),  and  head-up  display  (HUD).  Training  support  for  the  AIT  is 
provided  by  the  Student  Instruction  Control  Station  (SICS)  which  augments  the  real-time 
spatial  instrument  displays  by  providing  two-dimensional  graphical  presentations  of  the 
intercept  as  it  unfolds  in  time,  and  a  post-mission  feedback  system — AIT  DEBRIEF. 

The  SICS  graphical  display  consists  of  two  orthographic  projections^  indicating  the 
spatial  relations  of  the  aircraft  (color  coded  as  friendly  or  hostile)  and  the  ownship’s  radar 
scan  envelope.  The  orthographic  projections  depict  the  plan  view  of  a  40  x  20  mile 
airspace,  and  an  elevation  view  (35,000  ft)  of  the  same  airspace.  Together,  these  displays 
indicate  the  absolute  positions  of  all  aircraft  in  three-dimensional  space.  Although  each 
aircraft  is  identified  only  as  a  circle,  a  tick  mark  on  the  circle  designates  the  plane’s  heading 
track  vector. 


Figure  2 

The  Air  Intercept  Trainer. 

The  VirDI  system  interfaces 
with  the  AIT,  a  part-task  trainer 
for  concentrated  practice  on  F-16 
symbology  and  switchology. 


’  In  principle,  the  device  could  interface  with  any  training  system  capable  of  time  sampling  spatial  information 
during  real  or  simulated  flight.  The  system  can  accept  any  data  input  in  standard  ASCII  format. 

^  In  orthographic  projection  the  projection  plane  is  positioned  in  front  of  the  object  or  scene  so  that  the  line  of 
sight  is  co-incident  with  a  principal  axis  and  orthogonal  to  the  projection  plane.  All  projectors  from  the 
object(s)  are  parallel  to  the  line  of  sight  and,  therefore,  perpendicular  to  the  projection  plane.  The  traditional 
top/front/side  views  of  orthographic  projection  align  two  of  the  three  principal  axes  with  the  projection  plane 
and  collapse  the  third  axis  (the  line  of  sight  axis)  to  expose  two-dimensional  profiles  of  the  visible  surfaces 
(top,  bottom,  front,  back,  left  or  right). 


The  post-mission  feedback  system,  AIT  DEBRIEF,  has  been  operational  since  1991 
(Anselme,  1991).  This  feedback  system  contains  two  components,  one  for  data  capture 
and  a  second  for  performance  analysis.  The  data  capture  component  samples  mission 
performance  on  the  AIT  throughout  the  simulated  mission.  The  raw  data  set  represents  53 
variables,  time  sampled  at  a  rate  of  1  Hz.  Among  these  data  are  the  positions  and 
orientations  of  all  aircraft  (up  to  5  targets),  as  well  as  ownship  radar  information.  The 
performance  analysis  component  of  the  AIT  DEBRIEF  again  displays  a  pair  of  orthographic 
projections  indicating  the  flight  trajectories  of  all  aircraft  involved  in  the  intercept  from  the 
plan  and  right  side  profiles  (Fig.  3).  The  timing  and  sequencing  of  critical  mission  events 
are  depicted  as  they  occur  during  the  scenario  with  color  codes  and  other  symbology  (e.g., 
path  color  is  keyed  to  radar  mode).  Two  additional  windows  display  the  corresponding 
information  from  the  radar  and  HUD. 

VirDI  interfaces  with  the  Air  Intercept  Trainer  using  the  existing  AIT  DEBRIEF  data 
capture  component.  Data  collected  during  the  simulated  sortie  is  transported  to  a  personal 
computer,  converted  to  the  ASCII  format  and  prepared  for  graphic  projection.  Of  the  53 
variables  collected  by  the  AIT  DEBRIEF  system,  7  pertain  to  i  le  spatial  position  of  the 
ownship.  Twenty  additional  variables  describe  the  absolute  position  of  the  target  and  the 
position  of  the  target  relative  to  the  ownship  (6  additional  bits  of  information  are  collected  if 
the  radar  is  locked  onto  a  target).  Finally,  twenty  variables  describe  the  radar  mode  and 
scan  pattern.  This  data  set  is  truncated  in  VirDI  to  the  18  variables  listed  in  Table  1 .  This 
subset  of  data  describes  the  absolute  location  and  orientation  of  the  ownship  and  target, 
time-sampled  at  one-second  intervals.  In  addition  to  this  finely  grained  description  of  the 
flight  trajectories,  the  VirDI  data  set  also  represents  radar  mode  and  scan  pattern 
information.  These  data  represent  critical  intercept  information  to  be  displayed  as  they 
occur  along  the  flight  trajectories. 

Following  the  data  processing  stage,  the  time-sampled  air  intercept  data  set  is 
projected  onto  the  helmet-mounted  stereoscopic  displays  at  an  average  rate  of  15  Hz. 

The  total  elapsed  time  between  flying  a  sortie  on  the  AIT  to  walkthrough  in  the  virtual 
environment  is  5-10  minutes,  depending  on  the  duration  of  the  flight. 


Figure  3 

The  AIT  DEBRIEF  Post-Mission 
Feedback  System. 

AIT  DEBRIEF  currently  consists  of  a 
pair  of  two-dimensional  orthographic 
projections  indicating  a  top  view  and 
an  elevation  view. 
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Table  1 .  VirOI  Post-Mission  Feedback  Variables  Extracted  from  the  AIT  Data  Capture 
Component. 


Ownship  Position/Attitude 

Latitudinal  position  (degrees) 
Longitudinal  position  (degrees) 
Altitude  (ft) 

Roll  angle  (degrees) 

Pitch  Angle  (degrees) 

Yaw  Angle  (degrees) 


Target  Position/Attitude 

(up  to  5  targets) 

Rag;  whether  target  is  active 
Latitudinal  position  (degrees) 
Longitudinal  position  (degrees) 
Altitude  (ft) 

Roll  angle  (degrees) 

Pitch  Angle  (degrees) 

Yaw  Angle  (degrees) 


Radar  Variables 

Radar  center  of  scan  (azimuth) 
Radar  center  of  scan  (elevation) 
RCP  setting:  radar  range  scale 
RCP  setting:radar  elevation  bars 
Current  selected  radar  mode 


TRAINING  CAPABILITY  OF  VIRDI 

By  design,  the  VirDI  system  capitalizes  on  the  principal  properties  of  a  virtual 
environment — immersion  and  interaction  in  a  world  of  intangibles.  There  were  two  primary 
goals  in  developing  this  system  as  a  spatial  awareness  training  medium.  First,  it  was 
intended  to  provide  a  tool  for  visualizing  the  pilot's  performance  in  conducting  an  air 
intercept.  Performance  is  defined  here  primarily  in  terms  of  the  spatiotemporal  properties 
of  the  intercept  flight  trajectory.  Therefore,  the  VirDI  system  is  designed  to  project  the 
three-dimensional  spatial  coordinates  of  the  flight  trajectories  (self  and  target(s), 
up  to  5  targets)  as  they  unfold  during  the  intercept  maneuver.  Critical  intercept  events  are 
displayed  at  the  time  and  place  of  their  occurrence.  Second,  the  VirDI  system  was 
intended  to  provide  the  user  with  supervisory  control  of  a  three-dimensional  visualization 
environment.  Sheridan  (1987)  has  defined  supen/isory  control  as  an  interactive  human/ 
computer  system  in  which  the  human  sets  the  initial  state  and  intermittently  adjusts  the 
incoming  information  while  the  computer  assumes  a  direct  role  in  interacting  with  the  task 
and  the  task  environment.  Thus,  it  represents  a  compromise  between  automatic  and 
manual  modification  of  the  environment  by  the  user.  In  the  VirDI  system,  this  is 
implemented  so  that  the  user  controls  only  those  aspects  of  the  environment  which  will 
promote  insight  or  understanding  about  the  spatial  problem  and  pilot’s  performance  during 
the  simulated  mission.  Direct  control  of  extraneous  degrees  of  freedom  in  the  environment 
has  been  delegated  to  the  computer. 

Visualization  Capacity 

World  Properties.  A  typical  sortie  in  the  AIT  environment  encompasses  approximately 
40  square  miles  of  air  space.  In  the  VirDI  system,  the  virtual  world  also  extends  40  square 
miles,  with  50,000  ft.  vertical  air  space.  This  virtual  world  is  scaled  to  the  real-world 
coordinates  of  a  room  measuring  10'  x  10'  x  8'(H).  The  observer’s  natural  eye  height  and 
interpupillary  distance  are  not  altered.  Thus,  the  observer  is  effectively  a  giant  in  the 
Lilliputian  world  of  the  air  intercept.  This  adjustment  allows  the  user  to  view  the  entire  air 
intercept  from  a  single  vantage  point  while  maintaining  sufficient  proximity  to  capitalize  on 
binocular  disparity  information  for  stereoscopic  depth  perception. 
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Figure  4 

Close-Up  View  of  the  Debrief  Aid. 

Here  two  F-1 8s  and  their  respective  flight  trajectories  plotted  as  streamers  in  space.  The  pilot  may 

viewthetrajectoriesfrom  the  vantage  point  of  eithercockpitormayassumeany  external  viewpoint. 

When  viewed  stereoscopically,  the  trails’  three-dimensional  spatial  relations  are  evident. 

The  virtual  world  of  the  air  intercept  space  may  be  instantiated  as  a  naturalistic  flight 
environment,  a  room,  or  as  an  abstract  data  display.  When  the  virtual  world  is  presented 
as  a  naturalistic  environment,  it  is  furnished  with  a  simulated  terrain  and  various  natural  and 
man-made  objects  (e.g.,  mountains,  airport,  and  buildings)^  (Fig.  4).  In  early  demonstrations 
of  the  naturalistic  environment,  we  learned  that  users  tend  to  believe  that  VirDI  is  presenting 
a  real-time  interactive  event,  rather  than  an  historical  record  of  a  past  event.  As  a  result, 
they  would  attempt  to  participate  directly  in  the  sortie  as  if  flying  a  plane  (i.e.,  they  did  not 
just  explore  the  environment  to  better  visualize  the  data  record).  In  order  to  dispel  this 


^  Because  the  commercially  available  components  used  in  its  development  have  achieved  only  moderate 
sophistication  at  providing  visual  realism  in  real-time  graphics  displays,  the  VirDI  system  cannot  produce  a 
high  fidelity  virtual  environment  in  which  the  user  will  experience  total  immersion.  Nevertheless,  users  have 
reported  that  the  displays  are  sufficiently  compelling  to  engage  their  imagination.  Moreover,  because  the  user 
can  modify  the  viewpoint  as  well  as  affect  the  spatial  relations  among  objects  in  the  virtual  world,  it  is  not 
difficult  to  appreciate  “presence"  of  the  self  in  this  computer-generated  graphical  world. 
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impression,  we  have  instantiated  the  virtual  world  as  the  room  in  which  the  system  is 
housed.  The  virtual  room  shows  the  floor,  walls,  and  furnishings  of  the  real  room  at  their 
correct  spatial  locations.  This  allows  the  user  to  walk  about  the  real  room  in  safety  while 
viewing  the  air  intercept  trajectories  in  the  virtual  room  (cf.  Figs.  5  &  7).  Although  the 
trajectories  are  scaled  to  the  room,  the  user  may  step  out  of  the  virtual  room  and  view  the 
data  record  from  a  distance,  not  unlike  viewing  the  traditional  “sand  table”  used  for  strategic 
planning.  Finally,  the  abstract  virtual  environment  immerses  the  user  in  the  world  of  a 
three-dimensional  data  display.  The  axes  of  Cartesian  space  are  visible  and  are  scaled  to 
render  more  explicit  the  metrical  relations  among  the  ownship  and  target(s).  This 
abstraction  of  the  virtual  world  can  be  used  to  stress  to  the  student  that  performance  data 
may  be  subject  to  scientific  analysis. 

Air  Intercept  Trajectory  Properties.  As  noted  above,  the  primary  purpose  of  this 
visualization  tool  is  to  enable  the  pilot  to  assess  intercept  performance,  where  performance 
is  instantiated  in  the  flight  trajectory.  The  flight  trajectories  of  the  ownship  and  target  are 
portrayed  as  “snail  trails”  or  “streamers”  projected  into  the  three-dimensionai  space  (Fig.  5). 
Each  trail  is  attached  to  its  respective  aircraft  and  is  color  coded  for  ease  of  identification. 
Models  of  the  pilot’s  ownship  and  target  are  projected  at  a  scale  far  exceeding  realistic  size 
relationships  in  the  virtual  world.  This  exaggerated  size  is  necessary  to  render  the 


Figure  5 

Close-Up  View  ofThree  F-16s  with  Their  Flight  Trajectories  Plotted  as  Streamers. 
This  virtual  world  represents  the  room  in  which  the  system  is  housed. 


Figure  6 

Two  F-16s  with  Their  FlightTrajectories  Plotted  in  aThree-Dimensional  Graphic  World. 
The  pilot  may  view  mission-critical  events,  such  as  radar  mode  and  scan  pattern,  in 
order  to  better  understand  the  effect  of  certain  strategic  maneuvers. 


relatively  small  aircraft  visible  in  a  full-scale  environment  of  this  size.  The  aircraft 
continuously  travel  their  paths  whose  profiles  remain  stationary,  suspended  in  time  and 
space.  Thus,  the  viewer  can  examine  the  spatial  relations  of  the  aircraft  locally  and  globally 
at  any  given  moment. 

Critical  Intercept  Event  Properties.  During  the  course  of  an  air  intercept,  the  pilot 
performs  certain  mission-critical  actions.  Among  these  events  are  changes  in  the  radar 
mode  and  radar  scan  envelope.  At  present  the  F-16C  AIT  allows  five  radar  modes;  RWS 
(range  while  scan),  TWS  (track  while  scan),  ACM  (air  combat  mode).  ULS  (uplook  search) 
and  STT  (single  target  track).  The  pilot  also  controls  the  direction  and  size  of  the  radar 
antenna’s  scan  pattern.  In  VirDI  an  optional  display  mode  permits  presentation  of  the  radar 
envelope  as  a  visible  wireframe  projection  attached  to  the  aircraft.  The  wireframe  envelope 
is  color  coded  to  represent  the  radar  mode  operational  at  any  given  time  during  the 
mission.  Its  size,  shape  and  orientation  specify  the  antenna  elevation,  range,  and  azimuth/ 
elevation  scan  pattern  respectively.  As  the  radar  mode  changes  during  the  course  of  the 
intercept,  the  properties  of  the  projected  wireframe  envelope  change  accordingly.  Thus, 
the  pilot  or  instructor  may  assess  the  strategic  use  of  radar  functions  during  the  conduct  of 
an  air  intercept  (Fig.  6). 

Supervisory  Control  in  the  Virtual  Environment 

The  second  goal  in  developing  the  VirDI  system  was  to  provide  the  user(s)  with  control  in  the 
virtual  world.  Unlike  most  visualization  tools,  the  VirDI  system  allows  the  user  to  enter  the  computer¬ 
generated  virtual  world,  to  explore  the  environment  through  movement  and  to  act  on  other  objects. 
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Viewpoint  Control.  The  VirDI  system  supplies  the  user  with  the  means  of  visualizing 
the  Intangible — a  set  of  flight  trajectories.  To  increase  opportunities  for  gaining  insight  from 
the  virtualized  spatial  relationships,  the  user  should  be  able  to  actively  explore  the  data 
set — assuming  a  new  viewing  angle,  zooming,  panning,  and  so  forth  at  will.  Using  either 
the  head-tracking  system  or  the  Spaceball  input  device,  the  user  may  assume  either  of  two 
viewpoints,  an  egocentric  or  an  exocentric  frame  of  reference.  With  the  egocentric 
viewpoint,  the  pilot  is  positioned  within  the  cockpit  of  the  ownship,  as  when  the  original 
mission  was  flown  in  the  AIT.  Unlike  the  original  simulated  flight,  however,  the  pilot  can 
re-experience  the  flight  with  the  target  visible  in  the  “out-the-window”  view.  Relinquishing 
the  egocentric  point  of  view  allows  the  pilot  to  assume  the  exocentric  frame  of  reference  of 
an  active  outside  observer.  Once  outside  of  the  cockpit,  the  pilot  may  move  anywhere 
within  the  three-dimensional  world  and  assume  any  orientation  relative  to  that  world.  This 
allows  the  pilot  to  examine  in  detail  points  along  the  trajectory  at  which  critical  changes  in 
the  spatial  relationships  of  the  ownship  and  target  occurred.  The  real-time  image 
generation  system  is  designed  to  project  imagery  appropriate  to  the  gaze  direction  as 
interpreted  by  the  helmet-mounted  head-tracking  system  or  the  Spaceball  (Fig.7).  In  order 
to  control  for  a  user  becoming  ‘lost’  in  the  virtual  world,  a  simple  button  press  returns  the 
user  to  home  base  in  the  coordinate  system 


Rgure7 

Expert  Performance  Models. 
Student  pilot  compares  his  flight 
performance  with  those  of  two 
expert  fighter  pilots  by  walking 
through  the  simulated  airspace. 
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Instmctor  Feedback— Visual.  A  well  known  problem  in  pilot  training  is  the  fact  that  It  is 
not  possible  to  establish  a  “correcf  intercept  for  the  host  of  initial  conditions  and  tactical 
considerations  a  pilot  will  encounter  in  an  intercept.  On  the  other  hand,  the  student’s 
performance  on  a  given  mission  should  be  considered  a  tangible  product  that  is  open  to 
review,  discussion,  critique  or  comparison  with  other  air  intercepts.  One  of  the  most 
exciting  capabilities  of  the  VirOI  system  is  that  it  allows  the  student,  instructor  or  supervisor 
to  study  the  performance  and  provide  visual  and  auditory  feedback.  Visual  feedback  can 
be  provided  in  a  variety  of  ways: 

1 .  Models  of  Expert  Performances.  The  Air  Intercept  Trainer  allows  the  user  to 
experience  identical  initial  conditions  for  an  air  intercept.  Under  these  conditions, 
the  instructor  or  another  expert  pilot  can  tly  a  comparable  mission.  The  expert’s 
performance  trajectory  can  be  projected  in  the  virtual  environment  simultaneously 
with  that  of  the  student.  The  student  then  has  the  opportunity  to  compare  his/her 
air  intercept  performance  directly  with  that  of  an  expert  pilot  (cf.  Fig.  7). 

2.  Models  of  Personal  Best  Performances.  In  training  athletes,  contemporary  theory 
holds  that  rather  than  modeling  the  expert’s  perfomnance,  the  novice  may  benefit 
most  from  modeling  one’s  own  personal  best  performances  (Andersen,  1993). 
Because  the  AIT  can  provide  identical  initial  conditions,  the  student  can  practice 
intercepts,  varying  and  refining  maneuvers  within  a  specific  perfonnance  arena. 
One  or  more  of  the  performance  trajectones  can  be  projected  simultaneously  in 
the  virtual  airspace  for  comparison. 

3.  Instructor  Toolbox.  The  pictorial  format  of  the  virtual  environment  platform  also 
can  be  considered  a  virtual  blackboard  on  which  the  instructor  can  interact  directly 
with  the  portrayed  trajectory.  By  marking  the  virtual  blackboard,  the  instructor  can 
‘post’  editorial  comments  for  the  student  to  study.  The  VirDI  system  provides  the 
user  with  a  toolbox  of  icons  which  can  be  positioned  in  the  virtual  world  using  the 
Cricket  3D  input  device.  These  icons  serve  as  editorial  notation  symbols  that  are 
keyed  to  verbal  or  written  commentary.  The  icons  and  their  positions  in  the  virtual 
world  may  be  saved  as  a  part  of  the  permanent  data  file  for  future  access. 

Instructor  Feedback — ^Auditory.  A  critical  component  in  any  debriefing  system  is  the 
direct  communication  between  student  and  instructor.  Therefore,  much  of  the  success  of  a 
virtual  environment  feedback  system  may  depend  on  its  ability  to  permit  unimpeded 
discussion  between  the  student  and  instructor.  At  preserrt,  the  post-mission  instructor/ 
student  discussion  occurs  with  one  person  viewing  a  standard  computer  monitor  while  the 
other  experiences  the  virtual  world  within  the  helmet-mounted  display.  Both  users  can  view 
the  same  imagery  simultaneously  with  the  viewpoint  controlled  by  the  head  movements  of 
the  person  wearing  the  helmet. 

The  helmet  also  is  equipped  with  a  stereophonic  headset  for  inputting  sound  while  the 
user  experiences  the  virtual  environment.  At  present,  this  capability  is  exploited  with  a 
simple  audio  cassette  to  provide  recorded  auditory  feedback.  This  capability  expands  the 
system’s  versatility  by  allowing  the  instructor  to  tag  commentary  to  the  icons  posted  on  the 
virtual  blackboard.  Thus,  the  student  may  take  a  self-guided  tour  through  the  virtual 
environment  noting  the  instructor’s  iconic  annotations  on  the  projected  flight  path  while 
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listening  to  the  recorded  messege  tKitn  the  visual  and  auditory  feedback  may  be 

saved  or  transmitted,  it  is  feasible  that  ih&imk  t«is  (Movide  feedback  to  students 
stationed  at  remote  sites  within  minutes  of  lecetvwig  tt*4>  %tiKient  perforrrtartce  data. 

Summary 

In  summary,  the  first  level  of  development  for  the  Virtual  Environment  L>et/«i»( 

Interface — VirDl — provides  the  user  with  the  ability  to: 

1 .  Walk  through  the  virtual  environment  of  the  air  intercept  space  while  viewing  a 
three-dimensional  pictorial  display  of  the  flight  trajectories. 

2.  Explore  the  flight  performance  from  the  egocentric  cockpit  view  or  from  any 
exocentric  point  of  view. 

3.  View  multiple  fli^t  trajectories  (produced  by  the  student  and/or  experts)  simultaneously. 

4.  Pick  and  place  editorial  icons  within  the  virtual  world  that  are  keyed  to  auditory 
commentary. 

In  addition  to  the  instructional  benefits  that  are  anticipated  with  the  implementation  of 
the  VirOI  system,  this  platform  should  provide  low-cost,  portable  flexibility  that  presently  is 
not  available  with  existing  debrief  systems.  These  include: 

1 .  A  permanent  feedback  record.  Typically,  instructor  feedback  is  provided  verbally. 
With  no  tangible  copy  of  the  debrief,  the  student  must  rely  on  memory  when  at¬ 
tempting  to  correct  errors  and  improve  performance.  With  the  VirDl  system,  visual 
and  auditory  feedback  can  be  saved  for  review  in  standard  formats  of  computer 
ASCII  files  or  audio  cassettes. 

2.  A  portable  feedback  record.  Because  the  feedback  record  is  in  standard  format,  it 
is  available  to  anyone  with  the  capability  to  download  an  ASCII  file  and  has  a  tape 
recorder.  Thus,  it  can  be  shipped  to  remote  sites.  This  could  allow  a  centralized 
system  for  evaluating  intercepts  practiced  by  remotely  based  personnel  at  minimal 
cost  and  without  requiring  personnel  to  travel  to  training  centers. 

3.  A  training  library.  The  data  storage  and  display  capabilities  of  the  VirDl  system 
provide  a  readily  accessible  platform  for  developing  a  library  of  air  intercept 
scenarios  for  future  study.  The  library  can  contain  exemplars  of  successful  intercepts 
flownbyestablishedexpertfighterpilots,  as  wel  as  exemplars  of  unsuccessful  missions. 
Because  the  playback  allows  alternate  viewpoints,  the  user  can  experier>ce  the 

as  the  pilot  did  originally  or  view  it  exocentricaily  to  study  successful  maneuvers  or  disasters. 

FUTURE  GROWTH  OF  THE  VIRDI  SYSTEM 

In  this  rapidly  evolving  area  of  computer  technology,  we  can  presume  that  future 
hardware  will  soon  exceed  the  processing  capabilities  of  the  present  XTAR  system.  In 
anticipation  of  these  developments  (and  success  of  the  present  system),  it  is  not 
unreasonable  to  consider  available  technological  enhancements  to  support  the  following 
second  generation  VirDl  functions: 
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1 .  Data  display  extensions.  The  most  important  extension  to  the  display  capacity  of 
the  current  system  is  the  addition  of  simulated  cockpit  instruments.  At  present, 
the  system  provides  only  radar  mode  and  scan  envelope  information.  However, 
users  consistently  request  the  radar  and  HUD  data  that  they  were  monitoring 
throughout  the  flight.  Direct  access  to  the  radar  and  HUD  data  would  be  highly 
beneficial  because  it  would  allow  the  pilot  to  study  the  information  that  affected  his 
decisions  simultaneously  with  the  outcome  of  those  decisions.  On  the  other  hand, 
contrary  to  our  expectations  fow  users  have  indicated  a  need  for  greater  realism  in 
the  virtual  world.  It  appears  that  users  are  willing  to  accept  a  fairly  "unrealistic" 
virtual  world  if  it  offers  task-relevant  Information.  This  would  suggest  that  in¬ 
creased  processing  capabilities  should  be  devoted  to  display  of  cockpit  informa¬ 
tion  and  to  the  display  of  mission-critical  events,  such  as  weapons  control,  G-load 
and  energy  state  of  the  aircraft. 

2.  Expanded  interaction  capability.  At  present,  the  user  can  pick  and  place  editorial 
icons  in  the  virtual  .environment  and  key  these  symbols  to  written  or  auditory 
feedback.  The  next  generation  should  include  the  ability  to  draw  and  write  in  the 
virtual  world  using  a  three-dimensional  version  of  today’s  tele-lustrator^.  Using  a 
finger  or  pointer,  the  user  could  point  to  a  maneuvering  error  or  draw  a  suggested 
trajectory.  Drawn  trajectories  then  could  be  re-flown  by  the  pilot  to  experience 
directly  the  correct  path. 

3.  Shared  worlds.  At  present,  the  VirDI  system  supports  a  single  helmet-mounted 
display.  Future  developments  should  allow  multiple  users  to  experience  the  virtual 
air  Intercept  space  simultaneously.  If  supported  by  separate  graphics  display 
systems,  each  user  could  have  autonomous  control  within  the  virtual  world,  as 
well  as  the  ability  to  interact  with  other  users  of  the  world.  For  example,  if  an 
instructor  and  student  are  interacting  in  the  virtual  world,  the  instructor  could  point 
out  an  error  and  draw  a  corrected  trajectory.  The  student  could  observe  the 
instructor’s  changes  from  an  exocentric  viewpoint  and  then  enter  the  cockpit  and 
experience  the  correction  first  hand. 

4.  A  virtual  world  network.  The  Aircrew  Training  Research  Division  of  the  Armstrong 
Laboratory  at  Williams  AFB  is  a  leader  in  the  design  and  Implementation  of  com¬ 
puter  simulation  networks  for  simulated  air  combat  training.  As  networked  com¬ 
munication  systems  continue  to  advance.  It  is  desirable  and  feasible  to  link  the 
VirDI  system  to  a  training  network.  TT>ls  could  offer  several  advantages  in  enhanc¬ 
ing  the  training  effectiveness  of  the  network.  First,  remotely  based  personnel 
could  be  trained  or  evaluated  by  instructors  located  at  a  central  location.  Second,  h 
muttiship  combat  scenarios,  instructors  could  assess  individual  or  group  effectiveness 
from  a  command  and  control  center.  Pilots  also  could  review  their  performances, 
individually  or  in  groups.  This  would  allow  friend  and  foe  to  describe  their  strate¬ 
gies  and  techniques  or  experience  each  other’s  point  of  view  during  the  engage¬ 
ment. 


*  The  tele-lustrator  is  a  digitizedllght  pen  used  to  draw/Write  on  a  computer  or  television  screen.  It  is  often 
used  in  television  applications  for  direct,  on-line  input  to  the  screen  (e.g.,  weather  reports  or  sports  play-by- 
play  illustrations). 
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RESEARCH  OPPORTUNITIES  WITH  THE  VIRDI  SYSTEM 

Virtual  environment  debrief  systems  are  only  in  the  initial  stages  of  development. 

While  there  is  considerable  speculation  that  receiving  post-mission  feedback  in  a  virtual 
environment  could  improve  spatial  understanding  in  novice  pilots,  this  has  not  been 
established  empirically.  Indeed,  there  is  considerable  evidence  to  suggest  that  merely 
providing  higher  dimensional  data  in  a  three-dimensional  display  is  no  guarantee  that  the 
portrayed  spatial  relationships  are  comprehended  accurately  (cf.  Ellis,  McGreevy  &  Hitchcock, 
1987;  Smith,  Ellis  &  Lee,  1984).  Others  have  suggested  (cf.  Wickens  and  Todd,  1990)  that  if 
these  displays  do  lead  to  enhanced  performance,  the  TtoveHy  factor'  may  iricrease  the  time 
needed  for  the  student  to  process  the  infomnation.  Finally,  the  choice  to  implement  a  specific 
training  environment  must  be  based  on  the  nature  of  the  task  to  be  learned.  When  the  display 
format  is  not  compatible  with  the  task,  it  may  cause  interference  and  increase  the  cognitive- 
processing  load  on  the  user  (cf.  Liu  &  Wickens,  1992;  Wickens  &  Andre,  1988). 

Therefore,  a  significant  component  in  the  development  of  the  VirDI  system  is  the 
evaluation  of  its  effectiveness  as  a  visualization  tool  for  training  novice  pilots  to  construct 
mental  models  of  the  air  intercept  space.  At  present,  training  evaluation  studies  that 
employ  the  VirDI  system  are  being  conducted  at  AL/HRA  at  Williams  AFB,  A2.  In  one 
series  of  studies,  the  goal  is  to  determine  whether  the  VirDI  system  can  lead  to  a 
demonstrable  improvement  in  novices’  performances  at  air  intercept  maneuvers  on  the  AIT. 
Other  research  compares  the  effectiveness  of  two-dimensional  orthographic  displays, 
three-dimensional  perspective  displays,  and  the  VirDI  system  by  training  novices  to  locate 
a  target  from  HUD  information  alone.  It  is  anticipated  that  the  results  of  these  studies  will 
offer  considerable  insight  into  whether  the  virtual  environment  interface  can  advance  the 
fratntng  of  novice  pilots  to  expertise  in  three-dimensional  spatial  problem  solving. 
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